Search for: All records

Photographer, curator, and former director of photography at the Museum of Modern Art (MoMA), John Szarkowski remarked in *William Eggleston's Guide*, "While editing directly from life, photographers have found it too difficult to see simultaneously both the blue and the sky." Szarkowski insightfully revealed a notable gap between general and aesthetic visual understanding: while the former emphasizes identifying factual elements in an image (the sky), the latter transcends mere object identification, viewing it instead as an aesthetic component--a pure expanse of blue, valued purely as a color block in visual aesthetics. Such distinctions between general visual understanding (detection, localization, etc.) and aesthetic perception (color, lighting, composition, etc.) pose a significant challenge for existing Multimodal Large Language Models (MLLMs) in comprehending image aesthetics, which is increasingly needed in real-world applications, from image recommendation and enhancement to generation. To fundamentally advance the aesthetic understanding of MLLMs, we introduce a novel dataset, PhotoCritique, derived from extensive discussions among professional photographers and enthusiasts, distinguished by its large scale, expertise, and diversity. Additionally, we propose a new model, PhotoEye, an MLLM featuring a language-guided multi-view vision fusion mechanism for understanding image aesthetics from multiple perspectives. Finally, we introduce PhotoBench, a comprehensive and professional benchmark for aesthetic visual understanding. Our model demonstrates significant advantages over both open-source and commercial models on existing benchmarks and PhotoBench. 

The Segment Anything Model (SAM) is a large-scale foundation model that has revolutionized segmentation methodology. Despite its impressive generalization ability, the segmentation accuracy of SAM on images with intricate structures is often unsatisfactory. Recent works have proposed lightweight fine-tuning using high-quality annotated data to improve accuracy on such images. However, here we provide extensive empirical evidence that this strategy leads to forgetting how to "segment anything": these models lose the original generalization abilities of SAM, in the sense that they perform worse for segmentation tasks not represented in the annotated fine-tuning set. To improve performance without forgetting, we introduce a novel framework that combines high-quality annotated data with a large unlabeled dataset. The framework relies on two methodological innovations. First, we quantify the uncertainty in the SAM pseudo labels associated with the unlabeled data and leverage it to perform uncertainty-aware fine-tuning. Second, we encode the type of segmentation task associated with each training example using a task prompt to reduce ambiguity. We evaluated the proposed Segmentation with Uncertainty Model (SUM) on a diverse test set consisting of 14 public benchmarks, where it achieves state-of-the-art results. Notably, our method consistently surpasses SAM by 3-6 points in mean IoU and 4-7 in mean boundary IoU across point-prompt interactive segmentation rounds. 

Abstract Despite global warming, the sea surface temperature (SST) in the subpolar North Atlantic has decreased since the 1900s. This local cooling, known as the North Atlantic cold blob, signifies a unique role of the subpolar North Atlantic in uptaking heat and hence impacts downstream weather and climate. However, a lack of observational records and their constraints on climate models leave the North Atlantic cold blob formation mechanism inconclusive. Using simulations from phase 6 of Coupled Model Intercomparison Project, we assess the primary processes driving the North Atlantic cold blob within individual models and whether the mechanisms are consistent across models. We show that 11 out of 32 models, which we call “Cold Blob” models, simulate the subpolar North Atlantic cooling over 1900–2014. Further analyzing the heat budget of the subpolar North Atlantic SST shows that models have distinct mechanisms of cold blob formation. While 4 of the 11 Cold Blob models indicate decreased oceanic heat transport convergence (OHTC) as the key mechanism, another four models suggest changes in radiative processes making predominant contributions. The contribution of OHTC and radiative processes is comparable in the remaining three models. Such a model disagreement on the mechanism of cold blob formation may be associated with simulated base-state Atlantic meridional overturning circulation (AMOC) strength, which explains 39% of the intermodel spread in the contribution of OHTC to the simulated cold blob. Models with a stronger base-state AMOC suggest a greater role of OHTC, whereas those with a weaker base-state AMOC indicate that radiative processes are more responsible. This model discrepancy suggests that the cold blob formation mechanism diagnosed from single model should be interpreted with caution. Significance StatementThe mechanisms driving sea surface temperatures over the subpolar North Atlantic to cool since the 1900s remain uncertain due to the lack of direct observations. Here, we use a temperature change decomposition framework to dissect the historical trend of surface temperature simulated in multiple global climate models. The models diverge on whether the subpolar North Atlantic cooling is induced by reduced ocean heat transport convergence or altered radiative processes. Notably, the importance of ocean heat transport convergence is influenced by the simulated base-state strength of Atlantic meridional overturning circulation and the Irminger Sea’s mixed layer depth. This finding cautions against concluding the cooling mechanism from a single model and highlights a need for ongoing observations to constrain AMOC-related climate projection in the subpolar North Atlantic. 

Abstract Sea surface temperature (SST) has been increasing since industrialization with rising greenhouse gases. However, a warming hole exists in the North Atlantic where SST has cooled by 0.4 K/century during 1900–2017. It has been argued that this cooling is due to a slowdown of the Atlantic Meridional Overturning Circulation (AMOC), and subpolar North Atlantic SST has thus been utilized to estimate AMOC variability. We assess the robustness of subpolar North Atlantic SST as a proxy for AMOC strength under historical forcing, abrupt quadrupling of CO₂, and a medium future emissions pathway, finding that AMOC's fingerprint on SST depends upon forcing scenarios. AMOC is important in warming hole development during significant warming periods, although SST may introduce uncertainties for AMOC reconstruction in stabilized regimes due to diverse forcing mechanisms and decadal variability. Our results caution against using SST alone as a proxy for AMOC variability—both on paleoclimatic and contemporary time scales.